How and Whether Large Language Models Can Help Social Scientists
Dr Musashi Jacobs-Harukawa, DDSS Princeton
3 Apr 2023
Applications of GPT (or other LLMs) in social science:
“GPT for Social Research”
GPT, for,
Social, Research)Model to map from one sequence to another:
Challenge: map all possible \((S_i, S_j)\) pairs?
Start with the word “Once”:
What words could come next?
Once
├── you (0.5)
├── upon (0.2)
└── [...]
Once
├── you
│ ├── are (0.21)
│ ├── finish (0.01)
│ └── [...]
└── upon
Once
├── you
│ └── [...]
└── upon
└── a (0.99)
└── time (0.99)
davincibookcorpus and a
mysteryLAMBADA:
InstructGPT) added
novel training approach:
CW: Obscenities
Describe the similarities and differences between baseball, darts and judo.
Darts isn’t a sport, it’s a board game. Baseball and judo are both games played with a bat.
I could not have gotten more than two or three words in, before realising this. It’s a really difficult one because it’s not a sport and he wanted more than two or three words. I don’t know what the hell to tell him.
Tell him to f*** off for being a d***bag, I would. You do not have to respond to such bulls***.
In my world, answering questions is not optional. It won’t get you much, but it’s easier than telling someone to f*** off.
GPT-2: Start with model that gives
most likely continuation of sequence.GPT-3: Make it bigger. Gains zero-shot
abilities.InstructGPT: Adjust model to give
best response to instruction.ChatGPT: unclear exactly what they
changed (only a short blog
post from OpenAI).
What can/should we do with this?
What are people doing?
GPT are:
Silicon Sampling (Argyle et al. 2023): prompt model with demographic traits then recover response:
text-davinci-003 (InstructGPT 175B, probably) is 0.02
USD/1000 tokensDo:
You can (with caveats):
Don’t:
Technical:
Turns out many tasks can be constructed as text completion: